Exploiting Temporal Sequence Structure for Semantic Analysis of Multimedia

Automatic deduction of semantic labels for audiovisual data requires awareness of context, which in turn requires processing sequences of audiovisual scenes or events. The representation of such sequences is important for semantic analysis tasks. Whereas, conventionally, sequences of specific short-duration event labels, often hand-annotated for learning detectors or classifiers, have been used, we propose a new technique for audiovisual event categorization in this paper, wherein units of audio and image scenes are discovered automatically from data in a likelihood-maximization process. We show how these units for audio and video, respectively called AUDs and VIDs, can be used to learn the salient characteristics of broad-category semantic labels without requiring explicit error recovery measures. Experiments with the MED-11 dataset show that AUDs and VIDs are better able to retrieve semantic categories from mixed-content data as compared to vector quantization-based systems and systems that use library-based descriptors. Index: multimedia analysis, semantic labels, unsupervised lexicon learning, audiovisual data retrieval